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INTRODUCTION 


The present is a critical, frustrating, and exciting time 
in the history of computer graphics. A dozen significant 
trends are developing in the same field of view. Several 
much-needed trends are not developing. The technology 
is changing so rapidly that four generations of graphical 
systems are in operation at once. ‘Computer graphics” 
means different things to different people. This is there- 
fore a difficult session to “‘introduce.”’ 

Perhaps the most important introductory remark to 
make about the session is that the papers that follow 
this one do not attempt to cover the topic of computer 
graphics. They are in computer graphics, not on it. In 
this paper, I shall not try to cover the topic, either, but 
J shall take a brief look at the over-all field and try to 
sketch out enough of a map to provide an orientation 
and context for the papers that follow. In the process, I 
shall unburden myself of a few deep convictions. 

There are three main efforts in computer graphics. 
The first is to improve the capability of graphical dis- 
plays to represent things and processes. This ca- 
pability extends into dimensionality, verisimilitude, 
complexity, and motion. 

The second main effort is to improve the interaction 
between men and computers through graphics, to im- 
prove graphics as a medium of communication. It is 
important to distinguish between communication and 
representation. Both involve languages, but not in the 
same way. 

The third main effort is to develop applications of 
computer graphics. Applications run the gamut from 
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computer-aided composition of printed pages to com- 
puter-aided design and dynamic modeling. 
All the foregoing pose problems of computer-system 
design and are energized and/or inhibited by economics. 
This paper will deal, then, in an introductory—and 
necessarily sketchy—way with representational ca- 


' pability, communication and interaction, applications, 
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computer-system design, and economics—all in the 
context of computer graphics. Because the art is long 
and the time fleeting, the paper will consist essentially 
of observations and assertions within those areas rather 
than reviews or analyses of them. 


Representational capability 


Although most computer graphics is limited to ar- 
rangements of points and line segments, together with 
perhaps a hundred elementary figures (characters), 
computers are of course capable of processing and dis- 
playing areal pictures replete with graded lightness 
(brightness) and varied hue (color). The work of 
Stockham! at the M.I.T. Lincoln Laboratory and the 
University of Utah illustrates the versatility of the 
computer in handling still pictures with gray scale. 
Stockham scans a low-contrast photograph into the 
computer memory, takes the logarithm of every light- 
ness coefficient, applies a la Ernst Mach a second- 
spatial-derivative operator to each neighborhood of co- 
efficients to yield a new coefficient for the central 
element of the neighborhood, then takes the antilog- 
arithm and displays the resulting picture. Detail not 
visible in the original stands out clearly in the pro- 
cessed picture. But the processing requires many 
seconds for a picture with a millon elements, and 
Stockham’s kind of picture processing is not kinematic 
or “on-line interactive” in the present state of the tech- 
nology. Nor does the computer ‘‘understand’’ the 
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picture in the sense of forming a structured model of the 
pictured scene in which the various objects and their 
parts are represented separately in a hierarchy. Thus in 
Stockham’s application we see an emphasis of high 
precision in representation and processing at a sacrifice 
of speed and structuredness. 

The work of Evans, Warnock,’ and their colleagues at 
Utah starts with a structured model in the memory of 
the computer, rather than a high-imformation-content 
photograph at the input scanner. They have developed 
methods of processing that yield a halftone-like picture, 
in color, with ‘‘hidden lines” suppressed—in a small 
fraction of the computer time that would have been 
required two years ago. At the present time, however, 
even with the Utah methods, one cannot process and 
display areal (e.g., halftone) pictures fast enough to pro- 
duce moving pictures in real time. He can go to slow- 
time frame-by-frame generation of film, or he can 
schematize, or both. 

The technology for handling schematized pictures and 
diagrams—made up of points, line segments, and char- 
acters—has advanced markedly during the last two 
years. Roberts? described some of the basic ideas, in- 
volving homogeneous-matrix representation and per- 
spective transformation. In the work that won them the 
prize at the Fail Joint Computer Conference, Robert 
Sproull and Ivan Sutherland‘ developed hardware that 
very rapidly transforms selected parts of a model in the 
computer’s memory to a line drawing on the cathode- 
ray screen, The hardware rejects all the parts of the 
model—a large two-dimensional drawing or a three- 
dimensional scene—that would not be seen through a 
given window, and it produces the magnifications or 
perspective transformations and the clippings or trim- 
mings at the window frame necessary to display just 
what would be seen through the window. It does all that 
fast enough to display 3000 lines flicker-free. The “3000 
lines” refers to what is actually displayed; there can be 
many more lines that that in the model. The hardware 
handles curved lines and surfaces. It works hand-in- 
glove with the Warnock hidden-line algorithm, which 
works on schematized (linear) as well as areal displays, 
and the hardware is applicable to, and will decrease the 
processing time required for, areal displays. 

The foregoing dealt with actually two-dimensional 
displays of two- and three-dimensional objects and 
scenes. There are interesting ways to make the displays 
perceptually, or even actually, three-dimensional. The 
method of calculating and presenting two slightly 
different ‘“‘stereo”’ views is well known. So are methods 
based on the ‘‘kinetic depth effect,” which present a 
single, changing view. When successive two-dimensional 


cross sectional images are projected in rapid sequence 
upon @ moving screen, the eye, because of its per- 
sistence, sees the composite three-dimensional image. 
See Traub.’ The computer can contro! the settings of a 
matrix of push rods to make a three-dimensional sur- 
face, a histogram rising from a plane. And, of course, it 
can synthesize a hologram. 

Of those and other approaches to 3-D, the most ex- 
otic appears to be one demonstrated by Ivan Sutherland 
and associates at Harvard last year. The computer has 
a model of a situation in its memory, and it receives 
signals from sensors that tell it where the observer is 
and in what direction his head is oriented. With the aid 
of hardware described earlier, the computer then dis- 
plays the part of the modeled situation that the ob- 
server would see. The display is projected from a small 
cathode-ray tube on the observer’s head through a semi- - 
silvered mirror into one of the observer’s eyes, and the 
observer sees the displayed part of the situation out in 
front of him. He can look around and ‘walk around— 
and always see what he would see if he were in the 
modeled situation. When I saw the demonstration, 
the hardware was not finished, and the situation was 
just an outline room with windows, a door, and a geo- 
metrical piece of statuary. Even at that, it was quite an 
adventure. Given situations defined by thousands of 
line segments, one could surely create some exciting 
experiences—and probably some very significant ones. 
As Sutherland has pointed out, the “physical” laws of 
the modeled world can be determined by the program- 
mer. They can change with time. They can even de- 
pend upon the observer’s behavior. Elsewhere, the 
possibility of creating a direct four-dimensional expe- 
rience has been discussed. See Licklider.® I shall not go 
through it again here, but let me say that it will be in- 
tellectually at least as exciting to perceive and explore a 
synthetic 4-D world as to perceive and explore a merely 
actual, merely 3-D moon. 

Most of the capabilities of representation described 
in the foregoing are characteristic at. present solely of 
expensive equipment and are regarded by many as 
esoteric. I shall say something about that under the 
heading, Economics, but for the moment let it direct our 
attention to the other end of the spectrum. Among 


‘the most significant developments as judged from a 


practical point of view, certainly, are computer setting 
of type, computer-aided preparation of diagrams and 
graphs for the printed page, computer-aided com- 
position of advertisements, computer-aided generation 
of films, off-line production of graphics (as with curve 
plotters and as with the Stromberg-Carlson 4020), and 
fast on-line display of alphanumeric data and text. 
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Communication and interaction 


It is of course very important in computer graphics 
to be able to represent things and processes accurately 
and in detail. I think it is more important to be able to 
represent them in proper structural organization. I 
think it is still more important for computer graphics to 
be an effective medium of communication. Good one- 
way communication is good, but good two-way com- 
munication—good graphical man-computer interac- 
tion—is better. 

Now, as the history of art shows, and as Huggins and 
Entwisle’ will emphasize, and as I hoped Don Hatfield 
would develop in depth in this session, the representa- 
tion with the greatest verisimilitude is not often the best 
means or medium of communication. Good graphical 
communication is more a matter of linguistic expression 
than it is of pictorial reproduction. Unfortunately, we 
in the computer world do not know much about the 
language, and unfortunately not many of us are trying 
hard to find out about it. Obviously, it does not have 
much to to with programming languages. It probably 
does have something to do with data structures. But 
mainly it is the art of expressing ideas through con- 
figurations and manipulations of signs and symbols. Let 
me call that art an essential extension of computer 
graphics and then leave the topic to Huggins and 
Entwisle. I like their approach to .it, insofar as the one- 
way language—computer to man—is concerned. 

In my assessment, however, communication is es- 
sentially a two-way process, and in my scale of values, 
interaction predominates over detail, gray scale, color, 
and even motion. In my judgment, the most important 
problem in computer graphics is that of establishing 
excellent interaction—excellent two-way man-computer 
-communication—in a language that recognizes, not 
only points, lines, triangles, squares, circles, rings, 
plexes, and three-way associations, but also such ideas 
as force, flow, field, cause, effect, hierarchy, probability, 
transformation, and randomness. Nowhere, to the best 
of my knowledge, is such interaction approached in a 
broad problem area at the present time. The nearest 
things to exceptions are the graphical computer-pro- 
gramming interactions of Ellis and Sibley? at RAND, 
the partly graphical, partly alphanumeric on-line 
augmentation of Engelbart’s® intellect at Stanford 
Research Institute, and the graphical explorations in 
architecture of Negroponte” at M.I.T. It is very frus- 
trating to me that five and a half years have elapsed 
since Sketchpad passed its milepost without bringing 
more progress in man-computer interaction at the level 
of ideas and concepts. 

It seems unlikely that work in computer-aided design 
of devices and structures will lead to the advances I am 


hoping for. Computer-aided design of complex systems 
or processes might do it. I like the name, “‘interactive 
dynamic modeling,’ for the art. I want to see de- 
velop a kind of combination of computcr-aided design 4 - 
la Sketchpad and computer-program simulation 4 la 
on-line GPSS or OPS—in which the modeler and the 
computer engage in a high-level graphical interaction 
to formulate and test hypotheses for the solution of 
difficult problems that are not amenable to straight- 
forward iogico-maihematicai formulation, that involve 
more synthesis than analysis, more discovery than 
proof. Well, it is not much to have the name. I wish I 
had the method, the language, the software, and the 
hardware. 

Man-computer interaction of course presupposes 
computer-input devices (as well as languages) to medi- 
ate the communication from man to computer, The man 
should be able to signal the computer in a natural and 
synergic way, as by pointing and marking with a stylus 
while enunciating control words or phrases. He should 
be able to write or draw on the same surface as the com- 
puter. I shall not develop this part of the discussion 
beyond saying that one can appreciate the problem 
after he has read the transcript of a good blackboard 
lecture and then later (and separately) seen photo- 
graphs of the blackboard. “Chalk plus talk” is an 
excellent medium. Neither chalk nor talk alone carries 
much information. The communication must therefore 
liein the coordination. 

Applications 

At present, it appears that three very different 
classes of application of computer graphics are success- 
ful: (1) routine applications in the field of publishing 
(e.g., Chemical Abstracts) that yield printed pages as 
primary output and at the same time retain the in- 
formation in computer-processible form for updating 
or secondary exploitation; (2) routine applications 
throughout business and industry (e.g., airline ticket- 
ing, display of stock quotations) that involve fast on- 
line display of alphanumeric data or text from com- 
puters; and (8) ‘‘cream”’ (and often highly nonroutine) 
applications in government and industry (e.g., military 
command and control, space-mission control, seismic 
prospecting for oil) in which the premium on being 
first or best is very high or in which the required results 
cannot be achieved any other way. The first two do 
not require sophisticated graphic capability. The third 
requires and can afford very sophisticated capability. 

If and when there exists and is available the kind of 
computer graphics I referred to as “interactive dynamic 
modeling,” computer graphics will become a part of 
thinking and problem solving and decision making 
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wherever those functions are carried out. Design will of 
course be a major application area: design of all kinds of 
systems and processes—space, urban, transportation, 
manufacturing, military—as well as devices and struc- 


use graphics in its ‘command and control.’”’ Applica- 
tions will abound in research and development and in 
medicine. But the application area par excellence will 
be education. Almost anything not involving muscular 
skill that needs to be explained and demonstrated can 
be explained and demonstrated best with the aid of 
‘interactive computer graphics. For the sake of brevity, I 
shall leave that as a simple assertion in need of proof by 
demonstration. 

The trouble is, all those applications that depend 
upon a significant augmentation of the human intellect 
(to use Engelbart’s phrase) demand a level of computer 
graphics as sophisticated as that required for the 
“eream”’ applications mentioned earlier—or even more 
sophisticated. 

The field is ready for and cultivating simpleminded 
applications in which computer graphics will do faster 
or less expensively things that can already be done 
without it, but the field is not yet ready to accept the 
challenge of “mind expansion.” That carries us to the 
economics again—which we shall examine very shortly. 


System design 


The prevailing idea in computer-system design for 
graphics is that graphical display places too heavy a 
demand for processing to be handled by the (or a) 
main, central processor and that there should, therefore, 
be a satellite graphical processor for each graphics con- 
sole or cluster of graphics consoles. At present, the 
satellite processor is usually a small general-purpose 
computer augmented by a display processor. 

The design of the satellite processor is itself an in- 
teresting problem. Is it to be programmed once and for 
all and viewed thereafter as just part of the hardware, or 
is it to be thought of as remaining a programmable 
computer? Is its display processor to do nothing but 
display, or should it be able to call display subroutines 
and perhaps handle the integer arithmetic of indexing? 
Sutherland and Myer" have observed the tendency for 
the display processor to evolve into a general purpose 
computer and then to need a subordinate display pro- 
cessor, and they have determined how and where to 
stop the potentially infinite regress. 

The aspect of system design that I think has not been 
thought through clearly concerns the division of tasks 
between the central computer and the satellite com- 
puters. There is a watershed: either all the satellites will 
be near the central computer, or at least one will be 


remote. If all are near, why should each satellite com- 
puter have a separate memory? That leads to a wasteful 
transferring of data from one memory to another. Why 
shouldn’t the satellite processor be precisely a display 
processor and address a block of the main memory? 

If the satellite is remote, of course it cannot address 
the main memory directly; it must have a memory of 
its own. If one satellite is remote, all the satellites 
should have memories of their own, for it is most im- 
portant to preserve homogeneity for the sake of simplic- 
ity. 

Now we come to Bert Sutherland’s Dictum: ‘‘Think 
Network!” Even if you plan to have all the graphics 
consoles in the same room as the main computer, treat 
them as if remote—because one day soon it will be de- 
sirable to admit a remote satellite into the system— 
or you will want to share software with a system that 
has remote satellites. 

The key problem then becomes the division of tasks 
and the specification of the interface between the central 
and the satellite computers. That problem must be 
solved in such a way that the satellite does not pester 
the central computer continually, yet the user should 
feel that he is interacting directly with the central 
machine. I doubt that there is a good solution. 


Economics 


There are of course two main kinds of cost: (1) the 
cost of the equipment; (2) the cost of developing the 
methods and preparing the programs. The bad thing is 
that for sophisticated computer graphics both are very 
great. The good things are that the hardware costs tend 
to drop rapidly, once they start dropping, and that, as 
soon as there is replication of hardware, the software 
costs increase much less rapidly than the number of 
hardware systems in which the software is used. That is 
elementary, but it is very important and evidently not 
well enough recognized. 

During the 25-year history of digital computing, the 
costs of arithmetic units, control units, and processible 
memories have halved approximately every two years. 
On-line terminals have not had such a “‘long”’ history— 
at least not in significant quantity—but it appears that 
some of the simple keyboard-and-cathode-ray-tube 
consoles that are coming into use in time-sharing sys- 
tems dropped from about $13,000 to about $6,500 in the 
year between the last two Fall Joint Computer Con- 
ferences. If some are priced at $5,000 in the exhibit 
area of this conference, it will suggest that, for a time, 
one class of equipment will be halving in cost each year. 

I do not want to make too much of such gross trend 
analysis, even though it does point to the source of 
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much of the magic of our field, and I certainly shall not 
base anything on a halving of cost each year. However, 
the question of optimal research lead time is a very im- 
portant question, and it needs an answer. When should 
we develop the kind of computer graphics that will 
most strongly augment the intellect? When should we 
start if we want to have it ready when it will be afford- 
able? In their paper, Huggins and Entwisle’ will call 
educational application of interactive computer graph- 
ics, based on time sharing, an “economic absurdity” 
and suggest that the facilities be used to generate ed- 
ucational films. Insofar as operations are concerned, 
Tagree with them. But what about research? 

As basis for a rough calculation, let me take a system 
that I think would make an excellent base for a com- 
puter graphics laboratory. A few months ago I deter- 
mined.the cost of a graphics-oriented PDP-10 computer 
with 256% words of 2.5-microsecond memory, 15 million 
words of disk storage, 16 consoles with storage-tube dis- 
plays, and quite an assortment of supporting equip- 
ment. It was about $700,000. If it were used 8 years, 
with an average of 10 consoles active over the 24-hour 
day, the cost per console hour would be $1. Double that 
to cover maintenance, operation, supplies, and over- 
head, and you have $2 per console hour to use as an 
argument that you could be within hailing distance of 
economic operation now if you had the programs. 

As I see it, the advances in methodology and soft- 
ware required to achieve the kind of computer graphics 
and the kind of interactive dynamic modeling I hope 
for will require about as much research effort and time 
as several laboratories have devoted to interactive 
computing since 1961. If we look ahead to 1977, and 
assume hard work in the interim, I think we can count 
on a fairly good methodological and programming 
capability. If the halving-every-two-years rule held, 
the over-all equipment cost would be about 12 cents per 
console hour in 1977. It need not be that low to support 
all the educational and personal applications that it 
should support. Other applications—in management, 
research, medicine, etc.—would have become economic 
earlier, of course. 

My conclusion—which is of course as tentative and 
open to bias as the foregoing estimates, and perhaps as 
simpleminded—is that now is the time to push forward 


with research in the areas of computer graphics that 
people call expensive, sophisticated, esoteric, and exotic. 
I think that they are the areas in which lie the real 
promises of significant improvement in our intellectual 
processes. If we do not push forward with research in 
those areas now, we shall find ourselves with a magic 
lantern that we don’t know how to rub. 


_ REFERENCES 


1 T STOCKHAM A V OPENHEIM R W SCHAFER 
Nonlinear filtering of multiplied and convolved signals 
Proceedings of the IEEE 56 August 1968 126-1291 

2 J E WARNOCK 
A hidden line algorithm for halftone picture representation 
Technical Report 4-5 May 1968 University of Utah 

3 L G ROBERTS 
Homogeneous matrix representation and manipulation of 
N-dimensional constructs 
Lincoln Laboratory Massachusetts Institute of Technology 
July 1966 

4 R F SPROULL I KE SUTHERLAND 

A clipping divider 

AFIPS Fall Joint Computer Conference Proceedings 33-1 

December 1968 765-775 (Published separately by the 

Evans-Sutherland Computer Corporation Research Park 

Salt Lake City Utah 1968) 

A C TRAUB 

Stereoscopic display using rapid varifocal mirror oscillations 

Applied Optics 6 1967 1085-1087 

6 JC R LICKLIDER 

Computer graphics as a medium of artistic expression 

273-304 in Computers and Their Potential Applications 

In Museums New York Arno Press 1969 

W H HUGGINS D R ENTWISLE 

Computer animation for the academic community 

AFIPS Spring Joint Computer Conference 1969 

8 TO ELLIS W L SIBLEY 
On the development of equitable graphics I/O 
IEEE Transactions on Human Factors in Electronics 
HFE-8 March 1967 15-17 
9 D ENGELBART W K ENGLISH 
A research center for augmenting human intellect 
AFIPS Fall Joint Computer Conference Proceedings 
33-1 December 1968 395-410 
10 N NEGROPONTE 
The architecture machine 
Forthcoming The MIT Press 1969 
11 I E SUTHERLAND T H MYER 
On the design of display processors 
Comm of the ACM 11 June 1968 410-414 


or 


~1 


